-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[plsql,tsql] Fix CPD being case sensitive in PLSQL and TSQL #4943
Conversation
pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrToken.java
Outdated
Show resolved
Hide resolved
pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrToken.java
Show resolved
Hide resolved
Generated by 🚫 Danger |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
I didn't use a language property to disable this behavior (like done in Apex) although that could be done.
That's ok. We already removed the property in Apex, as it usually doesn't make sense to have this configurable. It's defined by the language and the language is either case-insensitive or not.
Maybe we could mention this (the fact that a language could be case-insensitive) in https://docs.pmd-code.org/latest/pmd_devdocs_major_adding_new_cpd_language.html and the like?
pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrToken.java
Show resolved
Hide resolved
pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/ast/PLSQLParser.java
Outdated
Show resolved
Hide resolved
pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/ast/PLSQLParser.java
Outdated
Show resolved
Hide resolved
pmd-plsql/src/test/resources/net/sourceforge/pmd/lang/plsql/cpd/testdata/sample-plsql.txt
Outdated
Show resolved
Hide resolved
.../test/resources/net/sourceforge/pmd/lang/plsql/cpd/testdata/sample-plsql_ignore-literals.txt
Outdated
Show resolved
Hide resolved
Co-authored-by: Andreas Dangel <andreas.dangel@adangel.org>
…to issue4396-cpd-case-sensitive
* The default just returns {@link Token#getText()}. | ||
* Transformations here are usually normalizations, for instance, mapping | ||
* the image of all keywords to uppercase/lowercase to implement case-insensitivity, | ||
* or replacing the image of literals by a placeholder to implement {@link CpdLanguageProperties#CPD_ANONYMIZE_LITERALS}. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Up to now these things would have been done in the token filter, which is supported by both Javacc and antlr.
I'm a little wary of adding yet another mechanism that can overlap. The only reason to get something into the lexer itself (the way Javacc is implemented) is because the behavior should apply both to cpd and pmd, as it's inherent to the language. The way this is hooked into antlr languages it's only applicable to cpd.
Yes, the tokenfilter was a catch all that made for some complex implementations and would probably use some cleanup, but I'm unsure this is is.
I feel we need to have a better defined boundary / limit extension points. Once this is published we will have to support it. I'd love to see this behavior pushed into the lexer rather the token / token manager as is the case for Javacc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd love to see this behavior pushed into the lexer rather the token / token manager as is the case for Javacc.
Do you mean having this behavior pushed into the Antlr lexer, so that the Antlr parser also sees the normalized image? In that case I agree, it would be nicer. I can look into it.
I think overall using the token filter to do that is an abuse when the language itself applies specific normalizations. That's why I believe our Javacc extension point is justified, and we need a mirror extension point for Antlr.
Edit: I see now that on the line you commented on, the reference to CPD_ANONYMIZE_LITERALS should be removed, as this part should be exclusive to the CpdLexer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you got my intent right.
Try and push this down to the antlr lexer, the same way the Javacc one works. It will seamlessly apply to pmd and cpd, make it non optional (it's language behavior), and exclude it as an extension point.
Other cpd specific behaviors such as anonymize literals should be done in cpd specific code. Right now that is the token filter, but as we both agree, at some point that api will need to evolve.
PLSQLParserImpl is public which shouldnt be
Describe the PR
Since these are two different lexer implementations (Javacc and Antlr), there are two different solutions which use the same idea, which is to project the image of most tokens into uppercase. I didn't use a language property to disable this behavior (like done in Apex) although that could be done.
Related issues
Ready?
./mvnw clean verify
passes (checked automatically by github actions)